pytorch as fast
r/MachineLearning - [D] Why is PyTorch as fast as (and sometimes faster than) TensorFlow?
Since both libraries use cuDNN under the hood, I would expect the individual operations to be similar in speed. However, TensorFlow (in graph mode) compiles a graph so when you run the actual train loop, you have no python overhead outside of the session.run In PyTorch, you are in Python a lot due to the dynamic graph, so I would expect that to add some overhead. Not to mention the fact that having a static graph means you can graph optimizations like node pruning and ordering operations. But in many benchmarks I see online, PyTorch has no problems keeping up with TensorFlow on GPUs.